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ABSTRACT 


The Department of Defense (DoD) and commercial industry expend significant 
resources to recover from failed maintenance actions. One known factor is the strained 
communication link connecting designers to maintenance professionals. Current 
technology leverages the technical manual, in both paper and flat electronic form, for this 
link. Augmented reality (AR) offers the potential to mitigate this deleterious factor by 
maintaining or transforming information into a more palatable form. This research 
measured human precision and efficiency by comparing augmented reality cued (ARC) 
and traditionally cued (TC) maintenance procedures in five tasks designed to elicit 
absolute, cumulative, absolute referential, and complexity errors across both ARC and 
TC conditions. Results indicate ARC procedures are statistically more efficient for 
human precise placement tasks of small parts, while precision is roughly equal. The 
assembly task, analogous to an assembly procedure, is statistically both more efficient 
and more precise using ARC versus TC procedures. ARC procedures for small part 
placement and assembly tasks are more efficient, faster, and in most cases at least as 
precise. 
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I. INTRODUCTION 


A. RELEVANCE 

The foundation of this work does not expose a new problem, but rather one that has 
plagued all human activity since the dawn of time: communication. Our technological 
advances have outpaced our individual capacity to fully comprehend at a level of detail 
required to build, operate, and maintain assets. Our solution of professional specialization, 
while necessary, constructs significant fundamental informational barriers between our 
artists, designers, engineers, operators, and our maintainers. While the computer has vastly 
broadened and deepened human understanding, we have erroneously strived to teach 
humans how to communicate with a computer within its domain. Although critical in some 
fields, it is corrosive to many other professional endeavors. AR changes this dynamic, 
presenting information in a more palatable form. 

One may envision, for example, the difficultly to program a robot with a camera to 
precisely place an object upon a table. The program must be able to: 

• localize the robot within its environment 

• identify the table within its greater environment 

• describe the table and designate parts of the table for referencing 

• identify the object of interest, differentiating it from other objects 

• describe a method to place the object in the intended position 

While conceptually simplistic, precision requires more information such as: 
object’s orientation within the world and how the objects characteristics define its own 
orientation, placement of the object upon a specific table and table’s differences from the 
standard table, measuring method and tools, calibration of the measuring equipment, and 
the validity of the current calibration. These examples highlight the potential for 
miscommunication between designers and maintainer. 
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B. EVOLUTION OF MAINTENANCE COMMUNICATION 


While not linear by any means, we can summarize the progression of the designer 
to maintainer communication pathway. Conceptually, we can place milestones to mark its 
adaptations. In the past, inventor/owner/operators served as the foundation or cornerstone 
of technological innovation. The moment the inventor shared their invention necessitated 
clear and repeatable communication. How does one ensure others can build, operate, or 
maintain this technology without constant supervision? Enter apprenticeship, our first 
milestone. Inventors taught and trained, in the desire to address small-scale transfer of 
technology, a cadre of users and maintainers. As widespread deployment of technology 
scaled outside proximal access, apprenticeships become over burdensome. Written 
engineered procedures, the next marker, mitigated this deficiency to a great extent by 
transforming spatial tasks into verbal ones, while mass production symbiotically amplified 
its positive effect. Our last marker brings us to the electronic domain, where deployment 
of written procedures becomes cheap and accessible. As a general statement, this domain 
did not alter the format or method of displaying information to the maintainer. Technical 
manuals merely transitioned to electronic documents, where new search functionalities 
optimized work. One may click on a hyperlink and acquire requested information, in text, 
graphic, video, or audio form. The information does not know what the maintainer is 
working on, and cannot objectively facilitate proper quality controls, it is passive. The 
maintainer must know what they are maintaining, what, on this object, is different from 
original specification, and know the process of gathering validated and verified contextual 
maintenance information. Can AR disrupt this dynamic? It is time to place another 
milestone? 

C. SHOULD DOD AND INDUSTRY TAKE NOTICE? 

All Department of Defense (DoD) services acquire, operate, and maintain assets at 
significant taxpayer expense. Both DoD and industry span the entire spectrum from heavy 
manufacturing assembly to individualized repair services and individually desire 
operational superiority over competitors. A method to maintain or achieve operational 
superiority over a competitor leverages technology. This leveraging can, in turn, raise the 
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minimum personnel requirements to operate and maintain assets. An increase in asset 
complexity creates secondary and tertiary cost ripples. Obviously, initial training 
requirements must be able to cover new technologies, but what about personnel currency 
training requirements? Do you need to add more personnel to perform the same amount of 
maintenance? Do you need a better quality-assurance system? What about quality 
assurance training? Raw acquisition costs typically increase, as do refurbishment actions. 
Potentially, the reduced breadth of manufacturers able to produce the technology as well 
as limited procured assets, add risk. Costs have the potential to increase exponentially. 

Typical systems contain significant cost drivers originating from operations and 
support (Jones, White, Ryan, & Ritschel, 2014). This is the origin of AR’s return on 
investment. AR can present guided contextual information simplifying maintenance 
processes. This can potentially reduce error and system specific training tails at the same 
time increasing efficiency. 

Most importantly, objectivity redefines the concept of maintenance 
communication. Instead of maintainers pulling designated information in a subjective way, 
computer vision coupled with object recognition can offload that communication stream to 
an objective push model. Information presented in this way, to the maintainer, is no longer 
passive, it provides immediate context. AR, in its fully matured form, can identify objects 
with the potential to: 

• know the object you are working on, and with reach back, all changes to 
its original specification 

• identify unknown or unverified changes made to an object 

• provide real time contextual information, a set of tasks for the specific 
object under current scrutiny 

• identify the proper tool, and observe its use 

• know the order of removal or installation of parts 

• know the current step of a procedure 


3 



• know a part is out of specification (broken) 

• designate locations without traditional measurement tools 

• gather raw data not processed through the human cognitive domain to 
ingest into logistic processes 

The previous list is only a subset of AR’s potential to redefine the communication pathway 
to a more objective one. 

D. SCOPE OF EFFORT AND THESIS ORGANIZATION 

This thesis attempts to answer the following research question: 

To what extent does augmented reality cueing affect efficiency and precision of 
maintenance tasks as compared to written forms of communication (e.g., technical 
publications)? 

A counterbalanced experimental study using a pairwise comparison establishes the 
foundation for all analysis and conclusions. During the experiment, subjects performed a 
set of maintenance actions decomposed to an atomic level. Precision and efficiency are 
metrics of their performance. The difference between the locations of the subject’s part 
from a known perfect position is precision, while task completion times coupled with 
precision assess efficiency. Subjects performed tasks under two conditions: TC 
communication (technical manual) and ARC communication (Microsoft HoloLens). The 
population of interest encompasses military and government (civilian) maintainers or 
assembly professionals. 

1. Omitted Areas of Study 

The state of AR commercial off the shelf platforms is not central to the discussion. 
Influences of a product in development on the assessment of precision and efficiency is 
important as Chapter IV expands. 

The scoping of this thesis precluded a holistic look at AR’s effect on all 
maintenance and assembly tasks. Chapter II bounds the research focus. 
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Evaluating a realistic maintenance task presents a multitude of issues. Many DoD 
maintenance procedures are too lengthy and do not present themselves in a way that 
facilitates scientific rigor. Confounding variables present significant assessment 
difficulties. For example, if there are only four bolts to remove the subject may intuitively 
progress through the task. Acquiring military parts pose resourcing, complexity, and in 
some cases classification restrictions. Availability and cost of useable military parts 
complex enough to enable scientific rigor is prohibitive. 

Although enabling symbiotic technologies for AR, computer vision and object 
recognition are deliberately minimized. Microsoft HoloLens headset integrated with 
Unity’s Vuforia is the only object recognition used and is used to locate the fiducial marker 
(amplifying remarks Chapter III.E.3.b). Fiducial marker recognition aligns the virtual 
world within the real world. Experimental scoping constrained the use of computer vision 
and object recognition to reduce variability and possible confounds. Mirrored or high gloss 
surfaces present significant challenges for computer vision facilitated object recognition, 
for example. 

This thesis is in no way a judgment of the Microsoft’s HoloLens platform. Chapter 
ITT addresses the question of choosing an experimental platform. 

2. Thesis Organization 

The background presented in Chapter II, defines AR, presents a few examples of 
hands-free AR platforms and industry’s use of them, examines similar research, and 
provides a broad overview task analysis for this experiment. Chapter III contains the 
approach which seeks to address questions expected of the reader while bounding the 
application of precision within four bins. Chapter III also describes study implementation 
and frames the experiment while simultaneously presenting limitations. Chapter IV, 
Results and Analysis, explains the process of extracting precision and efficiency data and 
presents the raw results. Chapter V affords interpretation while Chapter VI concludes with 
implications and future work. 
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II. BACKGROUND 


A. AR DEFINED 

As in many fields distilling a concrete, encompassing, precise and yet generalized 
definition for augmented reality (AR) is difficult. Two definitions give an introduction into 
AR’s domain. Milgram, Takemura, Utsumi, and Kishino (1994) place AR along a spectrum 
where one end is the physical world and the other is completely virtual. The authors coined 
the term Reality-Virtuality (RV) encompassing the entire spectrum depicted in Figure 1. 


Mixed Reality (MR) 


Real 

Environment 


Augmented 
Reality (AR) 


Augmented 
Virtuality (AV) 


Virtual 

Environment 


Reality-Virtuality (RY) Continuum 


Figure 1. RV Continuum. Source: Milgram et al. (1994) 


Milgram et al. (1994) further characterize AR as “augmenting natural feedback to 
the operator with simulated cues” (p. 284). Azuma (1997), in his seminal work, defines AR 
as possessing three essential conditions: a leveraging of both the virtual world and the real 
world, operation in real time to include user input and output, and three dimensional 
alignment of the virtual within the real world. These definitions are foundational to this 
thesis’ experiment where subjects leverage simulated cues to precisely orient real objects 
within the real world. 

B. A SUMMARY OF HANDS-FREE AR PLATFORMS 

Carmigniani, Furht, Anisetti, Ceravolo, Damiani, and Ivokoic (2011) propose a 
taxonomy of AR displays. Scoped effort for this research focused on user specific 
augmentation and hand-free platforms, which precluded any augmentation projected 
directly onto real objects. Table 1. is extracted from Carmigniani et al.’s taxonomy. 
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Table 1. Advantages and Disadvantages of AR HMD. 

Source: Carmigniani et al. (2011) 

Type of Display HMD 


Techniques 

Video-see-though 

Optical-see-through 

Advantages 

Complete 

Employs a half-silver 


visualization control, 

mirror technology, 


possible 

more natural 


synchronization of 

perception of the real 


the virtual and real 
environment 

environment 

Disadvantages 

Requires user to wear 

Time lag, jittering of 


cameras on his/her 
head, requires 
processing of 
cameras video 
stream, unnatural 
perception of the real 
environment 

the virtual image 


In this research, we focus on hands-free activity and offer a brief glimpse into 
current industry efforts within the head mounted display subdomain of AR. The chosen 
platform is Microsoft HoloLens, a head mounted product that maps the entire physical 
environment to better affix virtual objects or holograms within (Microsoft, n.d.). The 
Microsoft team included spatial audio along with voice and gesture inputs that enable hands 
free operation. Magic Leap is another full throated AR platform leveraging light field 
technology, object recognition, audio, and inputs for voice, eye tracking, head direction, 
and gestures (“Magic in the Making,” n.d.). Epson produces a head mounted display with 
1GB RAM, 8GB of storage, 6 hours of battery life, various sensors including GPS and a 5 
megapixel camera (“Augmented Reality and Mixed Reality,” n.d.). DAQRI smart glasses 
uses 64GB storage, AR tracking and depth sensors as well as audio, with a 5800mAh 
battery (“DAQRI Smart Glasses,” n.d.). ODG most recent smart glasses, the R-9, have dual 
1080p displays, and dual 5MP stereoscopic and AR tracking cameras (“ODG Compare 
Products,” n.d.). Glass takes a minimalist approach presenting monocular information to 
aid assembly or production (“Glass,” n.d.). Vuzix Blade smart glasses have an 8MP 1080p 
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camera, audio and additional vibration notification (“Augmented Reality (AR) Smart 
Glasses for the Enterprise,” n.d.). 

C. AR IN MAINTENANCE OR ASSEMBLY 

For many years academia and the scientific community pushed AR down the path 
from concept to scientific endeavor. As with any technology, maturation of both core and 
enabling technologies allowed industry to transform concept into application. A few 
examples of AR applications elicit enough understanding of the domain even though many 
companies have AR products in development. GE Aviation in 2017 piloted an effort 
combining Glass Enterprise Edition, Upskill’s software package and Atlas Copco’s torque 
wrench to evaluate mechanics’ performance (“GE Aviation Successfully Augmented 
Reality in Maintenance,” 2017). GE also evaluated wind turbine technicians within this 
environment. Both cases improved efficiency. Boeing ("Boeing Tests Augmented Reality 
in the Factory," 2018) recognized the advantages of Microsoft HoloLens enabled factory 
floor assembly procedures inducing a “90% improvement in first time quality” (para. 3) 
while simultaneously saving 30% of time. Additionally, the company tested wire harness 
assembly with Glass producing a 25% reduction in completion time while minimizing error 
(Sacco, 2016, para. 23). Abraham and Annunziata (2017, para. 10) analyzed the current 
manufacturing manpower situation with respect to gaps. They contend this technology, 
though various evaluations, enhances output by 32%. 

D. SIMILAR RESEARCH 

A Boeing team, circa 1992, produced four research applications featuring 
manufacture and maintenance activities highlighting both strengths and weaknesses 
(Caudell & Mizell, 1992). The authors communicate their endeavor to produce a “heads- 
up, see-through, head-mounted display (HUDSET)” (p. 659) where users can facilitate 
work through augmented cued procedures. Although the authors’ AR definition does not 
line up precisely with Milgram et al. (1994) or Azuma (1997), they capture the core ideas 
of AR. 

Tang, Owen, Biocca, and Mou (2003) evaluated the performance of 75 

undergraduate university students spread equally through 4 treatments: a manual, 
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instructions on a flat display, instructions on a display worn on the head, and AR. 
Measurements comprised perceived mental workload, speed, and precision in the assembly 
of Duplo blocks. The authors depict Figure 2. to express group average error, illustrating 
the dramatic benefits of registered AR for precision. 
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Figure 2. Average Error. Source: Tang et al. (2003) 


Tang et al. (2003) focused on an assembly task where only discrete error is possible, 
because Duplo blocks only connect in a finite number of ways. Figure 3. illustrates the time 
advantage AR has over a printed manual. 



Figure 3. Completion Time (Average). Source: Tang et al. (2003) 

While Tang et al. (2003) state their hypothesis that AR has a time advantage over 
other conditions is not supported, and conclude that the printed manual condition is 
significantly slower than all other conditions. 
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Henderson and Feiner (2011) conducted a user study of military mechanics for an 
AR application with a comparison to text based instruction via an LCD display and a heads 
up display. The authors indicated AR correctly directed attention on task faster than the 
traditional method of LCD display. 

E. TASK ANALYSIS FOR ARC AND TC TASKS WITHIN EXPERIMENT 

It is important to frame the scope of the tasks the subjects performed within the 
experiment. This thesis is not a grand statement on all maintenance activities. The 
following analysis is as a mechanism to frame the results. It uses the Department of Labor’s 
granular list of knowledge, skills, and abilities, etc., published on the 0*NET’s website, 
from which most tasks can be broken down (“Browse by 0*NET Data,” n.d.). This thesis 
breaks the experimental tasks down into subtasks which are definable by the items on this 
list. This analysis uses abbreviated categories from this list, such as “CA5,” defined as 
“Cognitive Ability 5: Memorization”. Appendix F contains the entire list of descriptions. 

1. ARC Task Decomposition for Cumulative Error Task in Experiment 

The following is a list of steps required and their associated task requirements. 
Although this list is from the designed task, some subjects innovated and altered the 
procedure in various ways. Some grabbed all parts, placing one at a time, while one placed 
two at a time. The task is to place a small part in the workspace in position quickly and 
precisely. 

1. Start task - Say “begin”: CA5, CA10 

2. Read instructions: CA1, CA3, CA5, SA1, SA2 

3. Understand instructions: BS1, BS3 

4. Grab part: CA2, CA5, PHA2, PSA1, PSA2, SA1, SA2 

5. Place part on paper: CA2, PHA2, PSA2, SA1, SA2 

6. Rotate part: CA8, CA9, PHA1, PSA1, SA1, SA2, TS1 


7 . 


Translate part: CA8, CA9, PHA1, PSA1, SA1, SA2, TS1 
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8 . 


Repeat step 2 to 6 4 more times 


9. Complete task / Say “complete”: CA5, CA10 

2. TC Task Decomposition for Cumulative Error Task in Experiment 

The following is a list of steps required and their associated task requirements. 
Although this list is from the designed task for the experiment, some subjects did not follow 
the procedure in various ways. Some only looked at the picture (did not have to understand 
text), some did not grab pencil, some measured all vertical measurements first, some 
measured all horizontal measurements first, some measured both before placing parts, and 
some did not even mark the paper workspace. The task is to place a small part in the 
workspace in position quickly and precisely. 

1. Say “begin”: CA5, CA10 

2. Turn page: CA2, CA3, PHA1, PSA1, PSA2, SA1, SA2 

3. Read manual: CA1, CA3, CA5, SA1, SA2 

4. Understand instructions: BS1, BS3 

5. Understand measurement: BS1, BS3, CA6 

6. Grab pencil and ruler: CA2, CA3, PHA1, PSA2, PSA3, SA1, SA2 

7. Measure Horizontal for 1st part: CA2, CA3, CA4, CA5, CA6, CA7, CA8, 
CA9, PHA1, PSA1, PSA2, PSA3, SA1, SA2, BS2, TS1 

8. Mark paper for 1st part: CA2, CA3, CA9, PHA1, PSA1, PSA2, PSA3, 
SA1, SA2, BS2, TS1 

9. Measure Vertical for 1st part: CA2, CA3, CA4, CA5, CA6, CA7, CA8, 
CA9, PHA1, PSA1, PSA2, PSA3, SA1, SA2, BS2, TS1 

10. Mark paper for 1st part: CA2, CA3, CA9, PHA1, PSA1, PSA2, PSA3, 
SA1, SA2, BS2, TS1 
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11. Drop pencil and ruler: CA2, PHA1, PSA2, SA1, SA2 

12. Grab part: CA2, CA5, PHA2, PSA1, PSA2, SA1, SA2 

13. Place part on paper: CA2, PHA2, PSA2, SA1, SA2 

14. Rotate part: CA8, CA9, PHA1, PSA1, SA1, SA2, TS1 

15. Translate part: CA8, CA9, PHA1, PSA1, SA1, SA2, TS1 

16. Repeat 3 to 15 four more times 

17. Complete task / Say “complete”: CA5, CA10 

Table 2. on the following page, depicts the differences between the two conditions 
for the cumulative error task. Salmon color highlights a step, with the associated coded 
abilities or skills, only one condition requires. Green highlights steps that are identical 
across conditions. Yellow highlights a difference in scope, while purple highlights the steps 
that would need a much deeper analysis to expose more fundamental differences. 
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Table 2. Differences between ARC and TC Procedures for Cumulative Error Task 

in Experiment 


Augmented Reality Cued 

Traditionally Cued 

1) Start Task/Say “begin” 

1) Start Task/Say “begin” 


2) Turn page: 

CA2, CA3, PHA1, PSA1, PSA2, SA1, 

SA2 


3) Read manual: 

PHA1, PSA1, PSA2 

2) Understand instructions 

4) Understand instructions 


5) Understand measurements: 

BS1, BS3, CA6 


6) Grab pencil and ruler: 

CA2, CA3, PHA1, PSA2, PS A3, SA1, 

SA2 


7) Measure horizontal for 1 st part: 

CA2, CA3, CA4, CA5, CA6, CA7, CA8, 
CA9, PHAl, PSA1, PSA2, PSA3, SA1, 
SA2, BS2, TS1 


8) Mark paper for 1 st part: 

CA2, CA3, CA9, PHAl, PSA1, PSA2, 
PSA3, SA1, SA2, BS2, TS1 


9) Measure vertical for 1 st part: 

CA2, CA3, CA4, CA5, CA6, CA7, CA8, 
CA9, PHAl, PSA1, PSA2, PSA3, SA1, 
SA2, BS2, TS1 


10) Mark paper for 1 st part: 

CA2, CA3, CA9, PHAl, PSA1, PSA2, 
PSA3, SA1, SA2, BS2, TS1 


11) Drop pencil and ruler: 

CA2, PHAl, PSA2, SA1, SA2 

3) Grab part 

12) Grab part 

4) Place part on paper 

13) Place part on paper 

5) Rotate part 

14) Rotate part 

6) Translate part 

15) Translate part 

7) Repeat steps 2 to 6 four more times 

16) Repeat steps 3 to 15 four more times 

8) Complete Task / Say “Complete” 

17) Complete Task / Say “Complete” 


CAx: Cognitive Ability, PHAx: Physical Ability, PSAx: Psychomotor Ability, SAx: Sensory 
Ability, BSx: Basic Skills, TSx: Technical Skills 


Only 43 required steps complete the entire task under the ARC condition, while the 
TC condition requires 69. From this alone, we can infer that the TC condition will take 
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longer. This does not fully describe, with enough granularity, efficiency and precision 
differences observed between the two conditions. A more meaningful decomposition may 
expose elemental contrasts within the purple highlighted areas of Table 1. Understanding 
how to put a physical part within a virtual 2D wire mesh, for example, requires a different 
set of skills, knowledge, and abilities than reading a graphically enhanced technical 
manual. Armed with this analysis, we could potentially connect ARC’s efficiency and 
precision characteristics with the medium’s ability to communicate spatial information 
directly to the user in its most palatable form. 
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III. APPROACH 


A. WHY COMPARE ARC TO TC? 

Foremost, AR is a dramatic change to the communication pathway between 
designer and maintainer. Arguably, virtual reality poses similar communication 
improvements, yet VR appears significantly constrained to the training environment. AR 
affords the opportunity to inject information into the operational environment, or during 
real time execution of procedures on physical equipment. Figure 4. describes usefulness of 
the medium versus proximity to the work environment. The post work environment focuses 
on aspects of quality assurance, performance assessment, and review. 





AR 

1 l>VR 

^>Flat electronic instruction 

^>Hardcopy instruction 



Training Near Work On Work Post Work 

Environment Environment Element Environment 


Figure 4. AR VR and Technical Manual Usefulness Comparison. 


With this in mind, AR has the potential to reduce training requirements while 
improving quality of performance. The impact of this technology inject will influence how 
and what type of accession training, workforce strength and proficiency, and the 
maintenance and assembly processes themselves. The fact that AR allows personnel with 
less training and experience to perform at or near the level of better trained or more 
experienced person has the potential to significantly disrupt a wide range of jobs and 
industries. 

TC maintenance procedures dominate the entire maintenance domain across the 
DoD. Systems designed many years ago transitioned from paper format to a flat or semi- 
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flat electronic format. Leading-edge system deployments do not appear to embrace virtual 
reality or AR in a holistic manner. 

With any maturing technology, AR is approaching a feasible and deployable state. 
Alongside sister technologies of computer vision, head tracking, and localization, AR 
presents significant promise in suppressing cost across the entire life cycle, as discussed in 
the introduction. Any new technology must be subject to strenuous scientific rigor paving 
the foundation for informed acquisition strategies. 

Industry is leading AR system integration and acquisition across a wide spectrum 
of professional activity in corporations such as GE (“GE Aviation Successfully Augmented 
Reality in Maintenance,” 2017) and Boeing ("Boeing Tests Augmented Reality in the 
Factory," 2018). Scientific rigor facilitates the justification for emerging technology 
acquisition, and ensures the DoD acquires the right tool for the right job at the right time. 

B. WHY ERECTOR SET PARTS? 

Erector set parts are cheap, replaceable, and available. This is in stark contrast to 
DoD parts that may contain hazardous material, classification, or acquisition barriers. 
Erector set parts also enable multiple levels and alterable complexity where DoD parts are 
typically as is. 

Most importantly erector set parts enable extraction of continuous precision data. 
In contrast to the work done by Tang et al. (2003) where placement of physical objects are 
discrete, this experiment forces users to express precision in a continuous context. 

C. WHY MICROSOFT HOLOLENS? 

This AR platform is a member of a growing field of hands-free devices. From a 
maintainer’s perspective, hands-free is essential to optimize productivity. The HoloLens 
has three factors that led to its selection as the platform for the experiment. 
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1. Maturity 

Even though Microsoft HoloLens is still in development, it bears a fairly robust 
development suite leveraging Unity among others. A support structure and development 
forum exist online. 

2. Availability 

Commercial enterprises and government entities create applications for this 
platform. Specifically, the Naval Postgraduate School employs a cadre of developers who 
are readily accessible and skilled in AR and VR development. 

3. Virtual Anchoring and Stabilization Technique 

To establish a base to anchor all virtual images within the real world, this platform 
continuously scans and maps the environment. Using this map, one may then place virtual 
images within context to the environment. Internal techniques then stabilize objects 
anchored in this manner. Those techniques allow a user to leave and return to a workspace 
knowing their objects retained their real-world positioning. This platform is also capable 
of occlusion of virtual images by real objects, enhancing immersion. In contrast, other 
platforms who use computer vision only, must reacquire the anchor point and then replace 
the virtual object where it should be every time visual tracking is lost. 

D. DEFINING PRECISION THROUGH OBSERVED ERROR 

Maintenance tasks span a wide range of human activity, each lending its 
contribution to precision of a specific action. In this analysis, we examine the effect not the 
cause. We are less interested in the decomposition of human activity to place an object, but 
specifically interested in the resulting difference in precision arising from that human 
activity. Observation of offset from an ideal provides error expressed in absolute, 
cumulative, absolute referential and complexity terms. They represent a significant portion 
of the maintenance and assembly error spectrum applicable to this research. 
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1 . 


Absolute Error 


The design of task 1 facilitates the expression of absolute error, or difference from 
a known fixed physical reference. As the subjects complete the same exact task 5 times 
(each offset from 5 separate L shapes), we can extract an average. How does absolute error 
connect to maintenance/assembly tasks? Example: Mark a specific location (place an 
object, drill a hole). ARC and TC procedures produce absolute error. AR systems, even 
with perfect tracking may anchor the virtual world incorrectly, inducing a global positional 
error. 


2. Cumulative Error 

Task 2’s design forces the subject to produce cumulative error, or error built upon 
error. While this task starts from a fixed reference, each successive placement references 
the placement of the last. Horizontal difference of the last piece from a known perfect 
location is the measured cumulative error. Grid production (pattern to drill holes) is an 
example where maintenance or assembly tasks connect to cumulative error. A computer 
can technically cannot produce this error, since all measurements reference one origin. 
Arguably, this is not exactly the case, where one may attribute drift in tracking, or even 
drift of the anchor, as methods for AR to produce cumulative error. 

3. Absolute Referential Error 

Task 3 asks the subject to place 5 objects a dissimilar offset from 5 L shapes. This 
activity forces the subject to reference their procedures to ensure correct placement. Every 
TC maintenance or assembly procedure forces professionals to understand information, 
memorize, transport to the work environment, recall, and execute the procedure. While this 
distance can vary from inches to hundreds of feet, a distance exists. This error results in 
both a positional difference (precision) and a time differential (efficiency). 

4. Complexity Error 

Tasks 4 and 5 seek to elicit both positional and time differences through a difficulty 
spectrum. Task 4 instructs subjects to place objects in a pattern. Task 4 separates itself from 
tasks 1 through 3 by increasing the separation from initial reference while complicating the 
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tasks. This may induce interpretation errors. Positional differences, with respect to a perfect 
placement, express precision while a time comparison between conditions enables 
efficiency computations. Task 5 guides subjects through an assembly, step by step. 
Incorrect connections, parts, or placement assess as precision error as well as incorrect 
routing, direction, and looping of the wire. 

Language in general, spoken or written, is a difficult medium to communicate 
various types of tasks. As complexity increases, both within process and task, individual 
interpretation has an opportunity to enter. Professionals could take short cuts due to 
procedural length, or use a better method because they believe they are more informed than 
the designer, or simply fail to adhere to procedure due to lack of understanding. All of these 
elicit interpretation and potential error. 

E. EXPERIMENTAL DESIGN 

After consent, a demographic survey gathered pertinent information about each 
subject. All subjects performed two task sets under two conditions. Subjects acquired 
subject numbers sequentially. Odd numbered subjects performed ARC then TC procedures 
with even numbered subjects receiving treatments in the opposite order. The first task set 
is comprised of 4 subtasks that require placement of “erector-set” parts. The second task 
set is an assembly task comprised of “erector-set” parts. One condition presents the subject 
with TC procedures and the other ARC procedures. A post-test survey gathered opinion 
data while video recorded objective data. 

1. Experiment Setup 

The following is a list of essential parts. 

1. Microsoft HoloLens 

2. Software: HoloLens Assembly Tester Software. Source: (Heine, Johnson, 

& Lee, personal communication, March 23, 2018) 1 


1 The thesis author conceptualized and designed the software. The development team created the 
application. 
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3. “Erector set” parts: Meccano Maker System Tower Bridge 

4. Video camera 1: Canon Vixia HF10 

5. Video camera 2: Sony HDR-XR520 

6. Video camera stand: AmazonBasic 60-inch lightweight Tripod with bag 

7. Steel magnetic blackboard 18inches x 24inches 

8. Ruler: Shinwa Stainless Steel rule H-101A 150mm (model 13005) 

9. Protractor: GemRed 2 in 1 Digital angle ruler in 200mm length 

10. Command strips: Picture hanging strips 

11. Magnetic strip: 3M 300LSE 

2. Tasks 

Each subject performed the experiment at one of two nearly identical stations. The 
only difference between the two was the make of the camera used to record their actions. 

Each subject conducted two task sets in each of the two counterbalanced conditions. 
As they performed the tasks, a video camera recorded their actions. Video examination 
determined time required to place parts, precision and correctness of their actions, as well 
as explaining any discrepancies or outliers in data. Figure 5 depicts an experimental station. 
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Figure 5. An Experimental Station 


a. First Task Set 

Task 1 (absolute error) has the subject place 5 identical erector set parts a set 
distance from 5 L shapes on a paper within their workspace (Appendices A, B and G). 

Task 2 (cumulative error) has the subject place 5 identical erector set parts a 
distance from one L shape on a paper within their workspace (Appendices A, B and G). 

Task 3 (referential absolute error) has the subject place 5 identical erector set parts 
a distance from 5 L shapes on a paper within their workspace (Appendices A, B and G). 

Task 4 (complexity error) has the subject place 3 difference erector set parts in a 
pattern on a paper within their workspace (Appendices A, B and G). 

b. Second Task Set 

Task 5 (complexity error) has the subject assemble a larger object out of erector set 
parts and a wire (Appendix A). 

3. Conditions 

Subjects completed both traditional cued and augmented reality cued procedures 
during the experiment. 
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a. Traditionally Cued 

This condition seeks to replicate the traditional method that many maintainers and 
assembly professionals work with: a technical manual. Text and graphics present 
information to the user. A manual created for the experiment (Appendix A) approximates 
a DoD style technical manual and would be familiar to DoD maintenance professionals. 

b. Augmented Reality Cued 

The subjects observed virtual images via Microsoft HoloLens as visual guides to 
place the parts. The NPS logo anchored these virtual images on paper within their 
workspace (Appendices B and G). This anchoring provided the basis for a lightweight user 
interface and object placement. Spoken commands, or gestures facilitated task progression. 
As stated earlier, object recognition only anchored the virtual space within the real space 
once per task. It did not identify and classify each individual physical part. 

F. LIMITATIONS 

Various issues related to hardware and resourcing presented a less than optimal 
environment for this research. 

1. The Platform 

Microsoft HoloLens is currently in development, which means that it is not 
commercially available and can change at any time. Inherent to the design are a couple of 
factors that impact this thesis. 50cm is considered the minimum effective range for 
comfortable viewing for relatively static environments, with virtual objects being removed 
at least 30cm away from the user (“Hologram stability,” n.d.). The website further states 
that the optimal working range is 2 meters (6.2ft). In this experiment, subjects are typically 
no farther than 55cm from the objects. Modifications to the HoloLens software displayed 
objects within 30cm by shifting the near clipping plane within the experiment’s AR 
application. 

Microsoft HoloLens can use a continuous mechanism to anchor holograms upon 
computer vision tracked real world objects. This method produces an image that could shift 
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without user movement. While not a problem for large-scale activity, this slight movement 
is problematic in the assessment of precise placement of objects. The experiment helps 
resolve this issue by anchoring each individual task once into the real world and turning 
off continuous updates. 

When AR systems anchor a 3D virtual world within the real world, a 6 degree of 
freedom global absolute error is born. Without future refinement this error will impact 
measurements pertaining to and placement of all virtual objects. Although AR systems 
traditionally can only present absolute error from anchoring, its drift would constitute 
cumulative error. Additionally, the uncorrected drift due to tracking errors, can likewise 
impact precision. 

2. Subjects 

Volunteers were subject to the following exclusion criteria: 

• Subjects unable to train to an unsupervised level of ruler and protractor 
proficiency after a maximum of thirty minutes of personalized training 

• Inability to see clearly without the use of reading glasses or bifocal/ 
cumulative corrective lenses (these corrections caused either a blurry real 
object and a well formed virtual image or vice versa) 

• Monocular vision (due to stereoscopic effects). 

3. Computer Vision 

An extremely limited use of computer vision using object recognition anchors the 
environment. A fiducial marker, an NPS Logo, specifically aligns the virtual world within 
the real. The Vuforia software element within Unity is intentionally further limited to initial 
anchoring of each task only. This prevents a jittery object or drastic shifts of the object. 
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4. Population of Interest 

The population of interest is government and military maintenance professionals or 
those who perform maintenance actions. For example, this would include pilots who 
perform minor maintenance actions in the performance of flying duties. 

5. Video Cameras 

Video cameras are the primary source for data extraction. With this medium, 
limitations spawning from fidelity, optical distortion, and multiple cameras arise. 

a. Resolution 

The video cameras used in the experiment have a resolution of 1920 x 1080. This 
resolution makes the measurements less accurate than a smartphone or a 4k camera, yet are 
good enough. 

b. Camera Differences 

Two video cameras produced data individually. Although the models differ, images 
bore the same resolution 1920 x 1080. A unique image distortion correction algorithm for 
each camera normalizes both optical workspaces for proper comparison. 

c. Optical Distortion 

Two issues arose relating to measuring true distances on images. First, fisheye 
effect bends the image further from the image’s center. Second, non-orthogonal image 
capture distorts measurements, similar to how objects look smaller at greater distances. 
Individual camera distances are static for this experiment. Chapter IV.C.l describes each 
correction. 


6. Two Investigators 

Based on throughput requirements, two investigators conducted the experiment. 
Knowing potential differences in presentation and emphasis could skew data, a script, 
shown in Appendix C, ensured conformity. Multiple training sessions reduced differences. 
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G. LIMITATIONS IN THE APPARATUS 

Using two different cameras is not optimal. It should be evident, even with 
individual corrections for each camera, a single model is ideal. Each camera exhibited 
slightly different zero zoom magnification. Although corrected for, by translating 
computations to the real world using with two different pixel densities, it again is not 
optimal. Given that resolutions at 1920 x 1080 were adequate for measurements to the 0.1 
of a millimeter, pixilation of the image added unnecessary ambiguity for determining 
measuring points. A video camera of much greater resolution would help resolve this issue. 

The calibration process attempts to alter the stereoscopic presentation to ensure the 
user observes the virtual image at the expected depth. Improper calibration results in 
parallax error as the image is closer or farther away than the intended depth. It is unknown 
how the internal calibration algorithm incorporates user input to compute interpupillary 
distance. Interpupillary distance is a necessary component for HMDs to adjust to each 
individual. Apparent subjectivity within the calibration process adds variability. 
Constructing a rigid calibration box could have eliminated this variability. 
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IV. RESULTS AND ANALYSIS 


Acquired data is in two formats: survey and video recorded human performance 
data. Survey data parsed into an Excel spreadsheet enabled subjective analysis while video 
data processed through Adobe After Effects facilitated precision and efficiency data 
extraction. 

A. BASELINING MEASUREMENT PROCESS 

Knowing the virtual image presented to the subject is in its exact specified real- 
world position by design, is very important to isolate human error vice computer 
compounded human error. 

The following is the process to ensure virtual images are in their intended real- 
world positions. The task’s virtual guides presented to the subjects during the experiment 
overlaid upon the task’s image created a simulated workspace. This workspace printed on 
11x17 paper formed the basis for baseline assessment. Comparing this printout to the 
normal printout of the subject’s workspace showed identical positioning and scale of all 
printed objects (NPS logo, L shapes, etc.). Real measurements of the L shapes (known to 
be 85mm high by 40mm) confirmed they are of identical length to the normal workspace 
sheets. Measurements with Adobe Photoshop then confirmed proper placement of virtual 
images within the real. 

Appendix H depicts the images that enabled the confirmation of perfect virtual 
object placement, by design. This does not infer object recognition, virtual to real world 
anchoring, or the tracking system of Microsoft HoloLens is perfect. 

B. OBTAINING EFFICIENCY DATA 

Video recorded human performance data pulled into Adobe After Effects facilitated 
analysis down to the hundredths of a second. Each subject had two sets of timing data: 
ARC and TC. 

Efficiency data is based on both timing and precision. In general, both longer 
completion times and more error are less optimal. More specifically, a smaller error for a 
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given completion time reflects better performance as does a quicker completion time for a 
given error. Efficiency data should also express numerically that a very precise action that 
takes longer would be as optimal as a less precise action completed very quickly. A simple 
algorithm of precision data multiplied by timing data meets the desired conditions. Smaller 
efficiency values are therefore better than larger efficiency values. This algorithm keeps 
the data unbiased and not beholden to a specific job. 

Different job requirements would alter this simple efficiency calculation. For 
example, a job that requires high precision and is less interested in completion times may 
increase the magnitude of influence for precision when calculating efficiency. A curve 
could represent this alteration giving increasing numeric importance to higher precision. 

The following groupings specify the extraction of performance measurements for 
this experiment. 

1. Tasks 1 through 3 

Subjects say “begin” to start their tasks. This marks the zero time of the task. The 
last touch or correction to the last placement of the erector set parts marks the end time. 
The differential is completion time. 

2. Task 4 

Subjects say “begin” to start this task. This marks the zero time of the task. The last 
time pencil marks scribe positioning guides is the setup time. A measurement from the 
setup time to the last part’s touch is pure part placement time, while the total completion 
time is measured from “begin” to the last touch or correction. 

3. Task 5 

Based on the complexity of this object, the zero time is at the word “begin” and the 
completion time is the time of last correction. 
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C. OBTAINING PRECISION DATA 

Examining the video of each subject determined the locations of the physical parts 
placed within their workspace. Aggregation of error from a known perfect position, 
measured in millimeters, is the precision data. Smaller results are more precise. 

1. Image Correction 

To achieve the required accuracy, it is essential to correct various image distortions 
originating from camera use. Both fisheye and non-orthogonality adjustments corrected 
image data sufficiently for proper data extraction. 

a. Fisheye Effect 

Adobe After Effects provides a function that corrects for camera fish eye image 
distortion. Focal length of the camera is a primary contributor to the correction. (Filter > 
Adaptive Wide Angle > Fisheye) 

b. Non-orthogonality 

An image generated from a non-orthogonally (lens axis not orthogonal to desired 
sample surface) placed camera produces a distorted image. Corrections are essential to 
ensure measurements based on the image are correct. Appendix D is a series of images 
validating the process. 

Given that image distortion may be non-linear we would not want to adjust the 
entire image at once. First, take the original image (Appendix D, Figure 44. ) and bound a 
smaller area to correct. Then distort the smaller area linearly (skew, distort, scale) to 
achieve a ratio that mirrors real space (known distances align). We know the F shape’s 
vertical line is 85mm and the horizontal line is 40mm. Appropriately scaling Adobe 
Photoshop measuring tool, so that it understands what 40mm is, now brings our 
measurements to a real-world scale. In this case, 567 pixels on the computer screen 
represents 40mm in the real world. An erector set part placed 10mm to the right and 
17.5mm above the F shape produced post-correction measurements of 9.84mm to the right 
(measured at the top), 9.84 to the right (measured at the bottom), and 17.46mm above the 
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L shape. The minor differences are well within the error of using a metric ruler and an 
image measuring tool, and not primarily attributed to the correction applied. Using this 
process, we prove that correcting non-orthogonality does not alter real world distances 
extracted from the corrected image. 

2. Task 1 

Two lines parallel with the horizontal portion of each L shape bisecting the center 
of each end hole of the erector set part and one vertical line parallel with the vertical portion 
of the L shape through the center of the bottom hole of the erector set part, become the 
measuring lines on the corrected sub-image. The measurement is from the inside of the L 
shape to the first pixel of the erector set along the measuring line. A summation of the two 
horizontal measurements gives the horizontal error, and the single vertical measurement is 
the vertical error. Absolute aggregation of each L shape’s vertical and horizontal error 
represents the total positional error of that part. Adding these absolute aggregations 
presents the total positional error for this subject on this task. 

3. Task 2 

Two lines parallel with the horizontal portion of the L shape bisecting the center of 
the end hole of the far-right erector set part and one vertical line parallel with the vertical 
portion of the L shape through the center of the bottom hole of the far right erector set part, 
become the measuring lines. Aggregate error is the horizontal error measured along each 
horizontal line added with the vertical error along the vertical line. 

4. Task 3 

This task is identical to task 1 with the exception of differing perfect positions. 

5. Task 4 

Task 4 has subjects place three physical parts: a part that looks like a C, a long part, 
and a square part. 
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a. C Part Precision 

Two perpendicular lines from the bottom line of the NPS logo to each bend of the 
C part that touches the workspace sheet, are guides for vertical error measurements. A 
measurement along each of these lines compared to the perfect placement’s distance 
creates two error values. The distance from the far-right vertical line’s intersection with the 
bottom of the NPS logo along the black line to the logo’s center is the horizontal 
measurement. Comparison to the perfect placement’s measurement creates the horizontal 
error. Aggregated vertical and horizontal error form a total positional error of the part. 

b. Long Part and Square Part Precision 

A nearly identical measurement process facilitates precision data extraction for 
these two parts. Vertical lines intersecting the center of the end holes of the long part 
establish the line of measure. The distance from the intersection of the NPS logo to the first 
pixel on the part along the vertical lines is the vertical position and its comparison to a 
perfect placement creates the error values. The distance from the far-left vertical line’s 
intersection with the bottom of the NPS logo along the black line to the logo’s center is the 
horizontal measurement. Comparison to the perfect placement’s measurement creates the 
horizontal error. Aggregated vertical and horizontal error form a total positional error of 
the part. The square part’s procedure is the same. 

6. Task 5 

Precision of this task expresses itself in in a binary way by misconnecting parts, 
missing parts, incorrect positioning of parts, incorrect direction of parts (bolts or wires), or 
misrouting of wires. The ARC task 5 has 54 possible correct actions: small bolt attachment 
point #1 (4 parts, one location, one direction), small bolt attachment #2 (4 parts, one 
location, one direction), C part (4 parts, one location, one direction), long bolt attachment 
(7 parts, one location, one direction), wire routing (27 possible correct actions of 
placement, direction, and looping). The TC task 5 has 52 possible correct actions: parts 
placement is identical to ARC, with only the wire routing possessing 2 less correct actions. 
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7. Validation of Precision Algorithm as a Metric 

A universally acceptable method to determine precise placement is not possible due 
to inherent subjectivity of the measure. Multiple biases exist that influence evaluation of 
precision. For example, an individual could erroneously determine hole alignment is the 
primary factor for precision because a task they performed in the past required it. 
Measuring precision must be objective and not influenced by bias. 

To resolve this problem, five people, who were not subjects in the study, ranked 
the precision of alignment of 5 physical shapes upon their virtual representation on a 
printed sheet of paper depicted within Appendix E. With this setup the consensus placed 
the fourth part from the left the most precise, then position 3, position 5, position 1, and 
finally position 2 as the least precise. The experiments’ precision algorithm agreed with 
the ranking of the human evaluators. 

D. RESULTS 

The counterbalanced experiment produced paired data from 34 Marine 
maintenance personnel. 16 subjects performed TC then ARC procedures and 18 subjects 
performed the opposite. Video corruption of 100% of a subjects’ ARC data has completely 
removed this one subject from paired analysis. This left the TC first group 16 subjects and 
the ARC first group 17 subjects. All analysis, histograms, and boxplots completed with 
JMP 13.1.0. Excel 2016 produced all graphs within this section. 

All analyses, that required a significance level, used an alpha of 0.05 unless 
otherwise noted. 

1. Efficiency Data 

Efficiency is work at a cost. In this context, precision multiplied by completion time 
is the algorithm of choice to compute efficiency. Millimeters of error represent precision 
and seconds represent completion time. 


34 



a. Task 1 (Absolute Error) 

Three subjects failed to complete the task, and one subject presented outlier data 
greater than three times interquartile range. This analysis did not include the subject, but 
Appendix I contains the original analysis with the subject’s data. Figure 6 is a histogram 
and boxplot, and Figure 7 is a paired t test of the remaining differential efficiency data. 




Figure 6. Histogram and boxplot of ARC minus TC Absolute Error Efficiency 
Data. ( x D = -8547.67 , s D = 9352.18, SE D = 1736.64 , 

95%C7 [-12105.04, —^4990.287], n = 29) 
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Figure 7. Raw ARC and TC Absolute Error Efficiency Data. Wilcoxon (TC minus 
ARC) results indicate that ARC is statistically more efficient than TC 
for absolute error (score x D = 28.97, SE D = 4.43 , z = 6.53, p < 0.0001, 

95%C7 [4313.24,10432.18]). 


b. Task 2 (Cumulative Error) 

Figure 8 is a histogram and boxplot, and Figure 9 is a Wilcoxon analysis on each 
pair of the remaining differential efficiency data. 
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Figure 8. Histogram and Boxplot of ARC Minus TC Cumulative Error Efficiency 
Data (x D =-8909.17, s D = 12242.32, SE D = 2131.11, 

95%CI [-13250.1,-4568.23], n = 33) 
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Figure 9. Raw ARC and TC Cumulative Error Efficiency Data. Wilcoxon (TC 
minus ARC) results indicate that ARC is statistically more efficient than 
TC for cumulative error (score x D = 32.67, SE D = 4.72, z — 6.91, 

p< 0.0001, 95%C/[3197.59,9930.9]) 
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c. Task 3 (Absolute Referential Error) 

Two subjects presented outlier data greater than three times interquartile range. 
This analysis did not include the subjects, but Appendix I contains the original analysis 
with the subjects’ data. Figure 10 is a histogram and boxplot, and Figure 11 is a Wilcoxon 
analysis on each pair of the remaining differential efficiency data. 
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Figure 10. Histogram and Boxplot of ARC Minus TC Absolute Referential Error 
Efficiency Data (x D =-5129.54, s D =5168.36, SE D = 928.26, 

95%C/[-7025.31,-3233.77], n = 31) 
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Figure 11. Raw ARC and TC Absolute Referential Error Efficiency Data. 

Wilcoxon (TC minus ARC) results indicate that ARC is statistically 
more efficient than TC (score x D = 30.58, SE D = 4.58, z = 6.67 , 

p <0.0001, 95%C7 [3009.2,5507.22]). 


d. Task 4 (Complexity Error) 

This task had two time elements extracted: total completion and placement only. 
Placement only time references only the time it takes to place objects on guides the subject 
previously placed. Two subjects failed to complete the task. 

(1) Total Completion 

Four subjects presented outlier data greater than three times interquartile range. 
This analysis did not include the subjects, but Appendix I contains the original analysis 
with the subjects’ data. Figure 12 is a histogram and boxplot, and Figure 13 is a paired t 
test of the remaining differential efficiency data. 
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Figure 12. Histogram and Boxplot of ARC Minus TC Complexity Error Efficiency 
Data (x D =-40608.15, s D =14995.14, SE D =2885.81, 

95%CI [^40540.03, -34676.28], n = 27). 
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Figure 13. Raw ARC and TC Complexity Error Efficiency Data (Total Completion 
Times). Paired t test (TC minus ARC) results indicate that AR is 
statistically more efficient than TC ( x D = 40608.2, SE D = 2885.8, 

t (26) = 14.07, p < 0.0001,95 %Cl [34676.28,40540.03]). 


40 




























(2) Placement Only 

One subject presented outlier data greater than three times interquartile range. This 
analysis did not include the subject, but Appendix I contains the original analysis with the 
subject’s data. Figure 14 is a histogram and boxplot, and Figure 15 is a paired t test of the 
remaining differential efficiency data. 


• • • 




Figure 14. Histogram and Boxplot of ARC Minus TC Complexity Error Efficiency 

Data ( x D = -1968.22, s D = 2397.94, SE D = 437.8, 

95%C7 [-2863.63,-1072.81]. n =30). 
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Figure 15. Raw ARC and TC Complexity Error Efficiency Data (Placement Only 
Times). Paired t test (TC minus ARC) results indicate that AR is 
statistically more efficient than TC (x D = 1968.22, SE D = 437.8, 

f(29) = 4.4957, p< 0.0001, 95%CT[1072.81,2863.63]). 


e. Task 5 (Assembly Complexity Error) 

One subject failed to complete the task and the software failed during one subject’s 
task. Figure 16 is a histogram and boxplot, and Figure 17 a paired t test of the remaining 
differential efficiency data. 
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Figure 16. Histogram and Boxplot of ARC Minus TC Assembly Complexity Error 
Efficiency Data ( x D =-87.21, s D =215.21, SE D =38.65, 

95%Cl [-166. 15, -8.26], 77=31). 
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Figure 17. 


Raw ARC and TC Assembly Complexity Error Efficiency Data. Paired t 
test (TC minus ARC) results indicate that AR is statistically more 
efficient than TC(x D = 87.21, SE D =38.65, f(30) = 2.26, ^ = 0.0315, 

95%C/[8.264,166.15]). 
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Precision Data 


Precision data is the aggregate error from a known perfect position along the 
horizontal and vertical axes. 

a. Task 1 (Absolute Error) 

Three subjects failed to complete the task and one subject presented outlier data 
greater than three times interquartile range. This analysis did not include the subject, but 
Appendix I contains the original analysis with the subjects’ data. Figure 18 is a histogram 
and boxplot, and Figure 19 is a paired t test of the remaining differential precision data. 
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Figure 18. Histogram and Boxplot of ARC Minus TC Aggregate Absolute Error 
(mm) (= 7.38, s D = 33.77, SE D = 6.27 , 95%C7 [-5.47,20.22], 

ft = 29). 
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Figure 19. 


Raw ARC and TC Aggregate Absolute Error (mm). Paired t test (TC 
minus ARC) results indicate that AR is not statistically different than TC 
(x D = -7.38 SE d = 6.27 , f (28) = -1.17 , p = 0.249, 

95%CT [-20.22,5.47]). 


b. Task 2 (Cumulative Error) 

Figure 20 is a histogram and boxplot, and Figure 21 is a Wilcoxon of each pair of 
the remaining differential precision data. 
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Histogram and Boxplot of ARC Minus TC Aggregate Cumulative Error 
(mm) ( x D = -24.53, s D = 42.57 , SE D = 7.41, 95%C7 [-39.63, -9.42]. 

n = 33). 
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Figure 20. 
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Figure 21. Raw ARC and TC Aggregate Cumulative Error (mm). Wilcoxon (TC 
minus ARC) results indicate that AR is not statistically different than TC 
(score x d =8.73, SE d =4.12, z = 1.85, p =0.0648, 

95%C7 [-0.12,34.2]). 


c. Task 3 (Absolute Referential Error) 

Figure 22 is a histogram and boxplot, and Figure 23 is a paired t test of the 
remaining differential precision data. 
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Figure 22. Histogram and Boxplot of ARC Minus TC Aggregate Absolute 
Referential Error (mm) ( x n = 14.21, s D = 27.59, SE n = 4.8, 

95%C7 [4.42,23.99], n = 33). 
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Figure 23. Raw ARC and TC Aggregate Absolute Referential Error (mm). Paired t 
test (TC minus ARC) results indicate that AR is statistically less precise 
than TC (x D = -14.21, SE D = 4.8, t( 32) = -2.95, p =0.0058, 

95%C7 [-23.99,-4.427]). 
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d. Task 4 (Complexity Error) 

Two subjects failed to complete the task. Figure 24 is a histogram and boxplot, and 
Figure 25 is a paired t test of the remaining differential precision data. 
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Figure 24. Histogram and Boxplot of ARC Minus TC Aggregate Complexity Error 
(mm) (x D =-5.81, s D =21.93, SE D = 5.02, 95%C7 [-16.05,4.43], 

ft = 31). 
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Figure 25. Raw ARC and TC Aggregate Complexity Error (mm). Paired t test (TC 
minus ARC) results indicate that AR is not statistically different than TC 
(x D =5.81, SE d = 5.02, ^(30) = 1.16, p =0.2557 , 

95%C7 [-4.43,16.06]). 


e. Task 5 (Assembly Complexity Error) 

One subject failed to complete the task. Figure 26 is a histogram and boxplot, and 
Figure 27 is a Wilcoxon on each pair of the remaining differential precision data. 
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Figure 26. Histogram and Boxplot of ARC Minus TC Aggregate Assembly 
Complexity Error (mm) ( x n = -0.02003, s D = 0.025 , SE D = 0.0044, 

95%CT[-0.02906,-0.011], n = 32). 
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Figure 27. Raw ARC and TC Aggregate Assembly Complexity Error (mm). 

Wilcoxon on each pair (TC minus ARC) indicate that AR is statistically 
more precise than TC (score x D = 25.53, SE D = 4.53 , z = 5.64, 


p <0.0001, 95%C7 [0.01,0.029]). 


3. Time 

This data represents the time in seconds from the beginning of precise motor skill 
activation to place a part in position to the end of the placement. Extracted data is for each 
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part within task 4 for both conditions. The analysis does not include one subject’s corrupted 
ARC data and associated paired data. 

a. Long Part 

Two subjects presented outlier data greater than three times interquartile range. 
This analysis did not include the subjects, but Appendix I contains the original analysis 
with the subjects’ data. Figure 28 is a histogram and boxplot, and Figure 29 is a Wilcoxon 
on each pair of the remaining differential time data. 
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Figure 28. Histogram and Boxplot of ARC Minus TC Time of only Precise Action 

(seconds) (x n = -3.506, s D = 3.064, SE n = 0.55 , 

95%C7 [-4.631,-2.382], n = 31). 
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Figure 29. Raw ARC and TC Times for Precise Action (seconds). Wilcoxon on 
each pair (TC minus ARC) indicate that ARC is statistically faster than 
TC (score x D =28.13, S£ D =4.58, z = 6.138, p< 0.0001, 

95%C7 [2.7167,4.183]). 


b. Square Part 

Figure 30 is a histogram and boxplot, and Figure 31 is a Wilcoxon on each pair of 
the remaining differential time data. 
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Figure 30. Histogram and Boxplot of ARC Minus TC Time of only Precise Action 

(seconds) ( x D = -3.157, s D = 4.763, SE D = 0.829, 

95 %CI [-4.845, -1.467] ,n = 33). 



Figure 31. Raw ARC and TC Times for Precise Action (seconds). Wilcoxon on 
each pair (TC minus ARC) indicate that ARC is statistically faster than 
TC (score x D = 21.697 , SE D = 4.73 , z = 4.591, p < 0.0001, 

95%C7 [1.783,4.223]). 
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c. C Part 

Two subjects failed to complete the task and one subject presented outlier data 
greater than three times the interquartile range. This analysis did not include the subjects, 
but Appendix I contains the original analysis with the subjects’ data. Figure 32 is a 
histogram and boxplot, and Figure 33 is a Wilcoxon on each pair of the remaining 
differential time data. 
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Figure 32. Histogram and Boxplot of ARC Minus TC Time of only Precise Action 

(seconds) (x D = 1.0967, s D =9.187, SE D = 1.677, 

95 %CI [2.334,4.527], n = 30). 
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Figure 33. Raw ARC and TC times for Precise Action (seconds). Wilcoxon on each 
pair (TC minus ARC) indicate that ARC not statistically different than 
TC (score x D =-6.3, SE D = 4.509, z =-1.397, p = 0.1624, 95% Cl [- 

3.016, 0.45]). 
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V. DISCUSSION 


A. HUMAN EFFICIENCY 

Completion time and precision combine in a simple algorithm to form efficiency 
data. Table 3 depicts the statistical advantage ARC has over every task within this 
experiment with respect to efficiency. 


Table 3. Efficiency Statistics 



Error Type 

95% Confidence Interval 

P Value 

Condition 

Advantage 

Task 1 

Absolute 

4313.24 

10432.18 

<0.0001 

ARC 

Task 2 

Cumulative 

3197.59 

9930.9 

<0.0001 

ARC 

Task 3 

Absolute 

Referential 

3009.2 

5507.2 

<0.0001 

ARC 

Task 4 

Complexity 
(Placement - 
Total Completion 
Time) 

34676.28 

40540.03 

<0.0001 

ARC 

Task 4 

Complexity 
(Placement - 
Placement Only 
Time) 

1072.81 

2863.63 

<0.0001 

ARC 

Task 5 

Complexity 

(Assembly) 

8.264 

166.15 

0.0315 

ARC 


Table 3 summarizes the results from Chapter IV, Section D.l, which shows that, 
for the reduced maintenance activity spectrum of this research, ARC procedures have a 
significant efficiency advantage over TC procedures. These findings are in agreement with 
the research of (Tang et al., 2003) and (Henderson & Feiner, 2011), and directly impacts 
various DoD and industry applications. A few examples could be: ship construction within 
drilling, cutting, welding actions which require some measurement to complete, rebuilding 
small complex objects made of many parts, wire routing during assembly of complex 
products, or even fixed intermediate level maintenance activity repair work. If object 
recognition is sufficient enough to properly align the virtual world within the real and 
identify specific parts, on-site maintenance is viable. 
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B. HUMAN PRECISION 


Various factors affect precision within this experiment. ARC procedures express 
precision from the AR hardware and software based on virtual world anchoring, tracking, 
calibration, and stabilization. Individuals’ sensory, perceptive, and cognitive abilities 
(SPC) and precise motor skills operate from that baseline as annotated in Chapter II, 
Section E. 1. TC procedures produce precision via physical measuring implements and their 
calibration, and the individuals’ SPC, and precise motor skills as described in Chapter II, 
Section E.2. The SPC loads of ARC and TC individual actions might be different. 

Table 4 summarizes the precision statistics for every task. Absolute error, 
cumulative error and complexity error (placement) shows no clear evidence of statistical 
difference. When only regarding precision, Table 4 shows that absolute error tends towards 
TC, cumulative error tends strongly towards ARC, and complexity (placement) tends 
towards ARC as advantageous. Statistically, it is very advantageous to use TC procedures 
to address absolute referential error and ARC procedures to address complexity error 
(assembly) when only regarding precision. 


Table 4. Precision Statistics 



Error Type 

95% Confidence Interval 

P Value 

Condition 

Advantage 

Task 1 

Absolute 

-20.22 

5.47 

0.249 

Neither 

Task 2 

Cumulative 

-0.12 

34.2 

0.0648 

Neither 

Task 3 

Absolute 

Referential 

-23.99 

-4.427 

0.0058 

TC 

Task 4 

Complexity 

(Placement) 

-4.43 

16.06 

0.2557 

Neither 

Task 5 

Complexity 

(Assembly) 

0.01 

0.029 

<0.0001 

ARC 


As a general statement, ARC procedures are not sufficiently different from TC 
procedures with respect to precision and small parts. When measuring absolute referential 
error, ARC was statistically less precise than TC procedures while ARC was statistically 
more precise than TC procedures during the assembly task. 
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Hardware, software, and human performance created this precision data. The 
precision data is not a general statement for all AR devices but specifically for both 
Microsoft HoloLens and the population of Marines that volunteered for this research. 
Additionally, the calibration routine did not appear optimized for close in work. As the 
technology progresses, precision will improve. 

C. INTERESTING FINDING 

Task 4, which directs the subject to place three parts within their workspace, has 
exposed a more fundamental difference than surface efficiency or precision comparisons. 
For the TC condition of the experiment, subjects completed a wire diagram first and then 
placed physical parts upon the diagram. This separated the measuring and drawing from 
the part placement action within the TC condition for this task. The ARC condition has no 
measuring or drawing component and prompts the subject to place parts directly. Both 
conditions’ placement only efficiency data, depicted in Figure 15, shows ARC is 
statistically more efficient while Figure 25, representing only precision, is not statistically 
different. These results precipitated another level of analysis captured within Chapter IV, 
Section D.3 which focused only on timing. All timing began from the transition to precise 
motor movements. With this more focuses look, ARC procedures are still statistically faster 
than TC for two out of the three parts. This necessitates a more granular determination of 
why the time to place a part guided by AR is statistically faster than placing a part upon a 
line an individual has already created. The SPC requirements between the two may be 
different for a similar task. For example, understanding where correct object placement 
guided by AR is different than a technical manual. AR may use a hologram of a wire mesh 
the size of the part vice text, numbers and pictures for the technical manual. This finding 
could be the result of different guidance structures presented by each medium. A much 
more granular experiment focused on this theory and analysis of the SPC requirements of 
each guidance structure may explain the difference. The author believes that the guidance 
structure of ARC placement tasks contains less magnitude and number of SPC 
requirements than the same motion outside of AR with a different guidance structure 
(lines). 
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VI. CONCLUSION 


A. IMPLICATIONS 

As industry advances the capability of AR platforms, the DoD must understand 
how AR technology can be leveraged to best advantage. This research scoped the spectrum 
of all maintenance activity to small part placement, an assembly task consisting of small 
parts, and wire routing. 

This thesis addressed the problem of efficiency and precision in maintenance tasks 
using ARC procedures hypothesizing that there is a statistical difference in both efficiency 
and precision when comparing AR to a technical manual. An experiment with 34 Marine 
maintenance professionals showed that ARC procedures are statistically more efficient for 
human precise placement tasks of small parts, while precision is roughly equal. The 
assembly task, analogous to an assembly procedure, is statistically both more efficient and 
more precise using ARC vice TC procedures. In conclusion, ARC procedures for small 
part placement and assembly tasks, including wiring, are more efficient, faster, and in most 
cases at least as precise. 

B. FUTURE WORK 

Given the wide breadth of procedural work and the potential impact small clarifying 
adjustments can have on the maintenance domain as a whole, it is essential to extend and 
expand research in this area. The following avenues are critical extensions: 

• Enlarge the size of the parts. Is there a definable curve or correlation to 
size of parts and human precision and efficiency? 

• Expand the sample types. How do other military services perform? 
Shipyard, automobile, and factory assembly professionals, are there 
differences? Does this research hold true outside of pure maintenance 
professionals? 

• Ambiguity of task. Is there a definable transition where intuitiveness, 

simplicity, or repetitiveness of a task overrides the benefits of ARC? 

61 



• Quality Assurance. To what extent does AR impact the correctness of 
maintenance actions? 

• Performance Metrics. What metrics are available to AR that is not 
available in other domains? Can these metrics better inform job 
performance? Can these metrics inform proficiency and proficiency 
decay? 

• Training. What impact does AR have on accession training? Is it possible 
some knowledge, skills and attitudes are less important when using AR? 

• Logistics. To what extent does AR enable more precise maintenance 
actions? Can AR’s objectivity help maintenance focus only on required 
repairs, not repairs based on time? 

• Calibration and Localization. What effect do these factors have on various 
ARC tasks? Is there a definable correlation between calibration and off- 
axis work? 

The author intends to investigate the unexplained advantage that ARC holds over 
TC with respect to placement of small parts when evaluating only precise motor 
movements. 

C. FINAL THOUGHTS 

ARC procedures, for small parts, require less steps to complete than technical 
manuals based on multiple factors. While procedures with less steps are generally faster, 
AR presents information in a more palatable form. This form appears to require a reduced 
number and magnitude of individual knowledge, skills and abilities for a similar precise 
action from a TC procedure. Is this the advantage AR has over traditional methods of 
communication? Specifically, the manner in which AR applications present information 
may be the fundamental and most meaningful advantage AR has over TC procedures. 
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APPENDIX A. TECHNICAL MANUAL 


A representative technical manual follows. The thesis author conceptualized, 
designed, and built this manual, including the procedures themselves. For formatting 
considerations, the manual begins on the following page. 
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ANGELOPOULOS SYSTEM TECHNICAL MANUAL 
CHAPTER 001 

VOLUME 1 - PROCEDURES FOR EXPERIMENTAL 
STUDY OF HUMAN EFFICIENCY AND PRECISON OF 
MAINTENANCE TASKS 



SUPERSEDURE NOTICE THIS MANUAL SUPERSEDES ALL PRIOR VOLUMES' 
AND ALL CHANGES THERETO 


1 Crest image Source: Department of the Navy (2011, August 1). NSTM 074 Gas Free Engineering Vol3 Rev 6. 
Washington, DC. retrieved from http://public.navy.mil 
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INTRODUCTION 


This port or of the experiment wrth guide you through 5 sections of tradit orally cuea 
procedures. You will perform base measuring, placement, and building actions Data from this 
experiment nforms future acqu siton decisons relatng to your field of expertise 


BACKGROUND 

The experiment is focusng on your individual abity to measure with BOTH precision arc 
speed. A ruler and a protractor afford the ability to place small objects. 


l|P S ge 
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TRAINING SECTION 

The first section wll refresh your memory for the use of a ruler arc a protractor. 

The max-mum tme for ths section is 30 ITI iflUteS 

You may ask questions or repeat work 

This is your tme to become comfortable wth the tools 

If you have not used a protractor, speno the majority of the tme doing that work. 
You do not need to complete all of the work 

When you are ready to begin: 

1 ) Say ‘Begin’ 

2) Turn the page 
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RULER Exerase: 


EL , 


1mm sioe 


D.5nr»m side 

Facts: The square eoge is zero, the smallest marts are 0.5mm. the ocher side's minimum is 


,L 


STEP 1: Using group of L_ shapes (first 2} and the ruler 

STEP la: Measure 5mm to the right of each vertical line, mark with a V. 


STEP 1b: Measure 8mm above each horizontal line, mark with an H*. 
PROTRACTOR Exercise 



• The intersection of the two inner edges must be placed on the vertex of the measured 
angle 

• Greater than 1 80 aegrees can be made by using the other side of the protractor 
STEP 2: Us ng the group & ^ shapes and the protractor 

STEP 2a: Measure ciockwse from the vert cal line Write down what angle is formed 
under the shape f ' 

Do this by turning on the protractor using the on/off button. Line up the straight edges 
like figure 1. Push the zero button Open the protractor clockwise (the direction is for 
the top straightedge) to roughly the correct angle. Line up the inner edge of the bottom 
straightedge with the vertical line (bear in mind the inner edge s the side of the 
protractor that forms the smaller angle). Place the vertex of the protractor on the dot of 
the line. Align the inner edge of the top straightedge on the other line. Read the 
number. 

STEP 3: Using the group of j shapes, measure with the protractor, indicated direction and 
number of degrees Make a new line from the center dot outward on the measured angle. 


"■■WfCN YOU FEEL CONFIDENT TO BE ABLE TO PERFORM THESE MEASURING 

ACTIONS: 

1) SAY'COMPLETE* 

2) TURN THE PAGE 
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SECTION 1 


Yoir role for this section is to: 

1) Place physical objects basec on precise measurements you make 

2) Proceed as fast as possible, but ensure you are stil placng the physcal objects precisely 
where you measure 

3) You can mark the worksheet as needed 

When you are ready 

1) Say 'Begin' 

2) Turn the page 
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Placement of 5 erector set pieces 


a) Positioned to the right of your workspace are 5 erector set peces Each piece has 4 
hoies in them. Use the ruler for the measurements 

b) One erector set piece will be placed nto each 'L' shape 


CAimON 


NOTE 


Do rot imosct the other erector SPEED AND PRECISION a*e 

set peces with your ruler when evaluated, 

measurirg. 

Step 1: Take an erector set pece placing it near the leftmost "L" shape 
Step 2: Orient the erector set pece so the longer edge s vertical 


Step 3: Measure 32mm to the right from the vertical Ine of the ‘L‘ shape. 

Step 4: Move the left edge of the erector set piece to fine up wth the measured distance away 
from and parallel with the vert cal line of the V shape 

Step 5: Measure 20mm up from the horizontal line of the 'L' shape 

Step 6: Move the BOTTOM CENTER EDGE of the erector set piece to Ine up with the 
measured distance away from and parallel with the horizontal line of the V shape 

"REPEAT THIS PROCESS TO FILL ALL THE V SHAPES WITH T>« OTHER ERECTOR 
SET PIECES" 



""WHEN YOU ARE COMPLETE 

1) Say‘Complete' 

2) Turn the page 


*!*»*■ 
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SECTION 2 

Your rote for this secbon is to: 

1) Place physical objects basec on precise measurements you make. 

2) Proceec as fast as possible, but ensure you are still placng the physcal objects precsely 
where you measure 


When you are ready: 

1) Say "BegrT 

2) Turn the page 
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Placement of 5 erector set pieces 

a) Positioned to the right of your workspace are 5 erector set peces. Each piece has 4 
holes in them. Use the ruler for the measurements. 

b) All 5 erector set pieces will be placed in a row start ng at the V shape 


CAimoh 

Do not to act the other e-ecto' 
set p eces with your ruler when 
measuring. 



Step t : Take an erector set piece placing it near the V shape 

Step la: Orient the erector set piece so the longer edge is vertical 

Step 1 b: Measure 12mm to the right from the vertical line of the V shape 

Step 1c: Move the left edge of the erector set pece to line up with the measured distance away 
from and parallel with the vert cal line of the V shape 

Step Id: Measure J6mn above the horizontal line of the V shape. 

Step 1e: Move the BOTTOM CENTER EDGE of the erector set pece to line up with the 
measure distance away from and parallel with the horizontal line of the V shape 


Step 2: Take an erector set piece placing it to the right of the last erector set pece 
Step 2a: Orient the erector set piece so the longer edge is vertical 


Step 2b: Measure :12mm from the right edge of the last erector part to the left edge of the new 
erector set pece 


Step 2c: Move the left edge of the erector set pece to line up with the measured distance away 
from and parallel with the last erector set part 


Step 2d: Move the BOTTOM EDGE of the erector set pece to line up with the previous erector 
set piece 

NOTf 


Ail 3ieces should be verbally 
parallel and horizontally in a line 


‘"REPEAT STEP 2-2d FOR ALL REMAINING ERECTOR SET PARTS 
■"■WHEN YOU ARE COMPLETE 

1) Say‘Complete* 

2) Turn the page 
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SECTION 3 


Yoir role for this section is to: 

t) Place physical objects basec on precise measurements you make. 

2) Proceeo as 'as: as possible, but ensure you are still placng the phys cal objects precsely 
where you measure 


When you are ready: 

1) Say'Begin' 

2) Turn the page 
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Placement of 5 erector set pieces 


a) Positioned to the right of yotr workspace are 5 erector set pieces Each pece has 4 
hoies in them. Use the ruler for the measurements 


b) One erector set piece will be placed nto each V shape 


CAUTION 


NOTE 


Do not impact the other erector SPEED AND PRECISION are 
set pieces vwth your ruler when evaluated, 

measurng 



Step 1 Take an erector set piece placing it near the leftmost 'L' shape. 


Step la: Place the left edge of the erector set piece 22mm right of the vertical line of the 'L' 
shape, and the BOTTOM CENTER EDGE 23m above the horizontal line of the V shape 

Step 2: Take another erector set piece placing it near the next - L' shape to the nght. 


Step 2a: Race the left edge of the erector set piece 21mm right of the vertical line of the V 
shape, and the BOTTOM CENTER EDGE 37m above the horizontal line of the V shape. 

Step 3: Take another erector set piece placing t near the next 'L' shape to the nght. 


Step 3a: Race the left edge of the erector set piece 26mm right of the vertical line of the V 
shape, and the BOTTOM CENTER EDGE 21 m above the horizontal line of the V shape 


Step 4: Take another erector set piece placing it near the next - L' shape to the light. 

Step 4a: Race the left edge of the erector set piece 33mm right of the vertical line of the V 
shape, and the BOTTOM CENTER EDGE 27m above the horizontal line of the V shape. 


Step 5: Take another erector set piece placing t near the next V shape to the right 

Step 5a: Race the left edge of the erector set piece 23mm right of the vertical line of the 'L' 
shape, and the BOTTOM CENTER EDGE 32m above the horizontal fine of the V shape 


""WHEN YOU ARE COMPLETE: 

1) Say "Complete* 

2) Turn the page 
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SECTION 4 

Your rote for this secbon is to: 

1) Place physical objects baseo on precise measurements you make. 

2) Proceec as ‘ast as possible, but ensure you are stil placng the physcal objects precisely 
where you measure 


When you are ready: 

1 ) Say "Begin" 

2) Turn the page 
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Placement of 3 erector set pieces 


a) Positioned to the right of yotr workspace are 3 erector set pieces 

b) You will place the erector set pieces to form a specific pattern. Use the ruler and 
protractor for measurements 


c) Mar king the she et is for your benef it ~hese line s are not evaluateo 


CAUTION 


NOTE 


NOTE 


Do not impact the other erector You may extend any line as SPEED AND PRECISION are 
set pieces when measuring needed to effectively use protractor evaluated. 



Figure 1: Overview of Part Placement. Source: (R. Lee. personal 
commtnication, February 26, 2018) 

Step 1: On the sheet, at the middle left side is an X. From the CENTER of the X, projecting 
inine along the 'X' per figure 2 below, (top left to bottom nght), draw a line i 09mm long. 


Step la: Mark the end of the line 'A'. 
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Step 2: Using 'A' as the vertex of the angle and the line described by the large X and the 'A', 
measure degrees ctocfrwise. per figure 3 below and draw a line 21.5mm long. 


Step 2a: Mark the end of the Ine B'. 



Step 3: Using B' as the vertex of the angle and the line descrbed by 'A' and B'. measure 90 



Mark the end of 


Step 4: Using C' as the vertex of the angle and the line labeled BC. measure 12c degrees 
courrfer-clockvwse. per figure 5 below, and draw a line ~mn long. 
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Step 5: Using 'D 1 as the vertex of the angle and the line described by 'C and 'D\ measure 3 
clockwise, per figure 6 below, and draw a line 25mm long. 

Step 5a: Mar* the end of the line 'E\ 



Step 6: Using '6' as the vertex of the angle, and the line labeled as BC. measure degrees 
counter-clockwise, per figure 7 below. Draw a line 75mm long. Mark the end of the line ‘F. 

Step 6a: Label the line 8F. 



13|Pa B e 


77 







A0001-AA-STM-001 


REVISION 0 


Step 7: Using 'F as the vertex of the angle, and the line labeled BF, measure 55 degrees 
counter-clockwise, per figure 8 below. Drama line 21mm long. Mark the end of the line ‘G'. 

Step 7a: Label the line FG. 



Step 8: Using 'G' as the vertex of the angle, and the line labeled FG, measixe degrees 
clockwise, per figure 9 below Dram a Ihe 4 5fT*r long Mark the end of the line W. 
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Step 9: Using 'H' as the vertex of the angle, and the line described by G and H. measure 
degrees counfer-ctocfcwise. per figure 10 below Draw a line 22mrr long. Mart the end of the 



Step 9a: Say the word 'FRAMED’ 

Step 10: Place the longest part, with its' end holes CENTERED over points 'B' and 'F. per 
figure 11 below 

Step 11: Place the square part with end holes CENTERED over points ' and ' ', per figure 
11 below 

Step 12: Place the double bent part, on its' side, with each 90 degree bend over points ’H" and 
T. per figure 11 below 

\ -l, 



Figure 11: Final Diagram. Adapted from: (R. Lee, personal communication, February 

26, 2018) 
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•"■WHEN YOU ARE COMPLETE: 

1) Say 'Complete' 

2) Turn the page 
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SECTION 5 

Your rote for this secbon is to: 

1) Construct a physical object comprised of erector set parts. 

2) Proceeo as fast as possible, but ensure you are stil connectng the physical objects precisely 
as indcated. 


When you are ready: 

1) Say "Begn" 

2) Turn the page 
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Step 1: Gather 4 parts: large rectangular box, one long straght part, one long bolt, one thick 
black plastic washer 

Step la: Feed the long bolt through the washer, through the last hole In the long part and 
through hole 5 of the large rectangular box from the right, as oepicted on figure 12 below. 

Step 1b: Hold parts n place and turn the page 



Figure 12: Step 1 & la. Source: (R. Lee, personal communication , 
February 26, 2018) 
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Step 2: Gather 4 parts: one long part, 2 thn siver washers and 1 silver nut 

Step 2a: Pe' figure 13 below, place the 2 thin silver washers, then the long part and the nut (in 
that order) or the long bolt from last step Tighten bolt while stabilizng nut (just enough to be 
snug I 
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Step 3: Gather 2 parts: 1 short bolt, 1 nut 

Step 3a: Per figure 14 below, place the bolt through both of the end holes of the long parts 
The bolt will be entering the same a rection as the long bolt (notce the figire has rotated 180 
degrees I. Place the nut on the other side. Tighten bolt (just snug). 

Step 3b: Turn the page. 



Figure 14: Step 3 Source: (R. Lee, personal communication , February 26, 2018) 
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Step 4: Gather 2 parts: t short bolt, 1 nut 

Step 4a: Pe r figure 15 below, insert the bcrt the OPPOSITE direction of the other bolts, and 
secure it with a nut Tighten bolt (just snug) 

Step 4b: Turn the page. 



Figure 15: Step 4. Source: (R. Lee, personal communication, February 26, 2018) 
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Step 5: Gather 3 parts: Double bent part 1 short bolt, 1 nut 

Step fa: Per fgure 16 below, find the hole (one up. one n) from the comer opposite side of the 
other attached parts 

Step fto: Insert bolt through center hole of double bent part, the appropnate hole in large 
rectangular part and secure the other see of the bolt with a nut Tighten bolt (just snug). 

Step 5c: Turn the page 



Figure 16: Step 5-5b. Source: (R. Lee, personal communication, February 26, 2018) 
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Step 6: Gather wre 

Step 6a: Feed entre wire through holes indicated in figure 17 below, with only 10mm left over 



Figure 17: Step 6a. Source: (R. Lee, persona/ communication. February 26, 2018) 

Step 7: Thread the re downward through ho*e 1 depicted or figure 18 below Bring the wire 
towards you and upwards until slack s taken out 


Step 7a: Thread the wre downward again throu^i hole 2 oepcted on figure 18 below It will 
wrap around the long part 


Step 7b: Turn the page. 



Figure 18: Step 7 & 7a. Source: (R. Lee, personal communication, February 26, 2018) 
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Step S: Gather ena toward you and feed the wre downward through hole 1 per figure 19 
below 



Figure 19: Step 8. Source: (R. Lee, personal communication, February 26, 2018) 


Step 9: Pufl the wire towards you. over the long part and aownward nto the hole per figure 20 
below 

Step 9a: Turn the page. 



Figure 20: Step 9. Source: (R. Lee, personal communication. February 26, 2018) 
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Step 10: Put the wire towards you and then outward through hole one then up through hole 2 
per figure 21 below 



Figure 21: Step 10. Source: (R. Lee, personal communication February 26, 2018) 


Step 11: Pull the wte upward, then feed the wire downward through hole 1 inccatec per figure 
22 

Step 1 la: Turn the page. 



Figure 22: Step 11. Source: (R. Lee, personal communication, February 26, 2018) 
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Final product This is what your object should look ifce 



Ftgure23 Final Outcome. Source: (R. Lee. personal communication. February 26, 

2018) 


""WHEN YOU ARE COMPLETE: 

1) Say 'Complete* 
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APPENDIX B. WORKSPACE SHEETS 


These figures represent the workspaces per task. To use them in the real world one 
must print them on 11x17 paper and ensure proper scaling. The thesis author 
conceptualized and designed these sheets. 



Figure 34. Workspace for ARC Training Task. Source: (R. Lee, personal 

communication, March 23, 2018) 
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Figure 35. Workspace for TC Training Task. Source: (R. Lee, personal 

communication, March 23, 2018) 



i 


Figure 36. Workspace for Task 1. Source: (R. Lee, personal communication, March 

23,2018) 
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Figure 37. Workspace for Task 2. Source: (R. Lee, personal communication, March 

23,2018) 
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Figure 38. Workspace for Task 3. Source: (R. Lee, personal communication, March 

23,2018) 
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Figure 39. Workspace for Task 4. Source: (R. Lee, personal communication, March 

23,2018) 



Figure 40. Workspace for Task 5. Source: (R. Lee, personal communication, March 

23,2018) 
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APPENDIX C. EXPERIMENT SCRIPT 


A. BEGINNING SCRIPT 

Thank you for volunteering to participate in this experiment. The experiment will 
take about one hour, but is not specifically bound by time. The experiment focuses on basic 
measurement and object orientation tasks performed through traditional methodology (a 
technical manual) and an augmented methodology (Microsoft HoloLens). Each 
methodology has a training session to ensure understanding that is open and not recorded. 
Questions are allowed during training. Once the experiment starts within each 
methodology, recording will begin, and very few questions are answerable. Be quick and 
precise with your actions. 

In front of you is a consent form you can begin reading and need to sign to 
participate. This information will be secured during and after the experiment. We will be 
recording your session, but only the workspace - not your face. All personally identifiable 
information will be removed from the video to the maximum extent practical. If for any 
reason at any time you wish to be removed from the experiment, just ask and the 
experiment will terminate. 

When you have completed the consent form please fill out the demographic survey. 
Upon completion we will begin the first training session. 

Key words will be used in the experiment, “begin,” “complete,” “next,” “framed,” 
“reset,” “select.” During HoloLens use the preferred method of selection is spoken, but you 
can use either gesture or speech. This will be covered at the appropriate time. 

Between each mini-session the experimenter will say “stand-by,” change the paper 
before you, and then say “when you are ready.” At that time, you may begin the next task. 
You don’t need to memorize these tasks, you will be reminded. 
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B. TECHNICAL MANUAL TRAINING SCRIPT 


This training session is intended to familiarize you will the use of a ruler and a 
protractor. It is at your own pace and not recorded. The 11x17 paper before you is the 
workspace. The workspace paper may be written on in any manner you desire to help you 
complete the assigned task. Please take care of the manual, which is face down at this time 
to your left, and do not mark within it. You may place it in any location that best fits your 
needs without blocking the camera or your access to your parts. Parts will eventually be 
placed on the right side of the workspace. 

When you are ready please turn over the manual and proceed. 

***** <AT END of TRAINING SESSION - MAX 30 MINS> “Do you feel able 
to use the ruler and protractor on your own,” IF NO: “We appreciate your time but we 
cannot include you within the study.” IF YES: We are now entering into the experiment, I 
will most likely not be able to answer questions. <START CAMERA RECORDING> 
When you are ready <POINT AT MANUAL> 

C. MICROSOFT HOLOLENS TRAINING SCRIPT 

This training session is intended to familiarize you with the use of Microsoft 
HoloLens™. It is at your own pace. You may repeat work as needed to become familiar. 
Let’s first cover the headset itself. In the back is a tightening/loosening dial. <SHOW> 

Do not overtighten, I repeat, do not overtighten, it should just be enough where it 
does not feel like it is falling off/loose. When donning the headset ensure this portion of 
the headset <POINT> is roughly parallel to the ground. If you are wearing glasses you 
may gently pull the outer ring forward <POINT> to get the lenses of the headset in front 
of your glasses. All in all, the optics should be in front of your eyes with the nosepiece on 
your nose. If the headset is misaligned, when you look through the optics it will feel like 
you are looking up or down to see the middle of the optical field of view. The center of that 
field of view is a small white dot. The small white dot is surrounded by a halo effect when 
on menus. When the dot is over a button it is highlighted. Turning your head to place the 
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white dot on a button, and then using a selection technique progresses you like a mouse 
and mouse button click would. 

Now let’s cover important HoloLens™ gestures. Consider gestures discrete actions. 
Being expressive will not help. 

The area in front of you roughly bounded by your shoulder width and % arm’s 
length is a viable area for the HoloLens <SHOW>. 

Using the area right in front of your eyes, while effective, is not necessarily optimal 
for hand movements <SHOW>. 

The first gesture is called bloom. It is performed in two steps. Hand in cone shape 
facing up. Then slowly raise hand while opening coned fingers. <SHOW> 

The next gesture is called a sky pinch. It is a way to select something similar to a 
mouse button click. Raise and hold open ‘pinch’ of forefinger and thumb for one second. 
Then close ‘pinch’ touching forefinger and thumb <SHOW>. Make sure you can see the 
pinch point of your fingers, if you cannot Microsoft can’t either. <SHOW BAD ONE, 
THEN GOOD ONE> 

We will use verbal commands with the HoloLens™. The word for sky pinch is 
‘SELECT’. It will select anything under the center white dot. 

“Begin” will be used to start each task. Before you say begin, ensure you are 
comfortably seated and centered on the NPS graphic in the workspace. Stabilize your head 
movement as you say begin. Move freely after the graphic changes. “Complete” will be 
used to end each task. “Next” will be used to proceed to the next step in a task. Lastly a 
very important command is Reset. It will repeat a step. With all of these commands a ‘sky 
pinch’ button will be available if you so prefer. 

Bear in mind when a 3D virtual object is presented before you, you are able to look 
around the object to get a ‘better view’. <SHOW> 

Are you ready to put on the headset? <HELP PLACEMENT> Be careful not to 
tighten too much. Can you see the white dot? <LOOK AT SCREEN ON TH E 
LAPTOPx Ok, let’s do a bloom. Now move your head to place the white dot on the plus 
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to the right. <VERBALLY WALK THEM THERE>. Sky pinch. Move your head to 
Calibration on the first row <SAY SELECTx Please follow the prompting. 

When complete: please bloom. Look at the plus sign. ‘Select’. Look at HoloLens 
assembly on the second column. ‘Select’. Look up. ‘Select’. <DIRECT THEIR VISION 
TO THE NPS LOGO>. 

<After they say begin and they are in process of placing the last object>: As 

you can see the text box presents information to you that you can reference. The buttons 
on the bottom or side can be actioned verbally with their names, or you may sky pinch 
them. Any questions about that? Do not expect to the see the entire workspace at once. If 
you cannot see a portion of the workspace, look around or back up a bit. A stop sign image 
is to the left of the disc image and closer to you. Can you see it? Does it look like it is on 
top of the symbol? <If they say yes and the placement of objects is way off redo 
calibration> 

Are you comfortable with what you have seen so far? <IF YES, TELL THEM TO 
VERBALLY ACTION THE ‘C’ WORD YOU SEE ON THE RED BUTTONx 

Stand-by. <SET NEW PAGE DOWN> <START CAMERA RECORDING! !!> 

<SET PIECES TO PROPER LOCATION> We are now entering into the experiment, I 

will most likely not be able to answer questions. When you are ready. 

****<SAY STAND-BY AFTER THEY SAY COMPLETE> <CHANGE THE PAPER> 
<SET PIECES TO PROPER LOCATION> <SAY WHEN YOU ARE READYx 

D. ENDING SCRIPT 

<AFTER LAST TASK: STOP RECORDING> <PLACE SURVEY IN 
FRONT OF THEM> Please take a few minutes and fill out the post-task survey. <WHEN 
COMPLETE> Thank you for your participation, Any Questions? 

Place both surveys within folder. Reset experiment for next participant. Check time 
left on SD memory card. If less than 1.5 hours then replace with new one. 
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APPENDIX D. IMAGE CORRECTION 


The thesis author conceptualized the image correction. A digital graphic designer 
developed the method using Adobe After Effects. 



Figure 41. Image Correction Template for Tasks 1 and 3. Source: (R. Lee, personal 

communication, March 23, 2018) 
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Figure 42. Image Correction Template for Task 2. Source: (R. Lee, personal 

communication, March 23, 2018) 



Figure 43. Image Correction Template for Task 4. Source: (R. Lee, personal 

communication, March 23, 2018) 
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Figure 44. Initial Part Placement within Template. 



Figure 45. Top Horizontal Measurement after Image Correction. Source: (R. Lee, 

personal communication, March 9, 2018) 
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Figure 46. Bottom Horizontal Measurement after Image Correction. Source: (R. 

Lee, personal communication, March 9, 2018) 



Figure 47. Vertical Measurement after Image Correction. Source: (R. Lee, personal 

communication, March 9, 2018) 
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APPENDIX E. PRECISION EVALUATION IMAGE 



Figure 48. Precision Evaluation Image. Adapted from: (R. Lee, personal 

communication, April 23, 2018) 
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APPENDIX F. Q*NET ABILITIES AND SKILLS 


The entirety of this appendix is verbatim from the website (“Browse by 0*NET Data,” 
n.d.) 

A. ABILITIES 

1. Cognitive Abilities 

Abilities that influence the acquisition and application of knowledge in problem 

solving 

• CA1: Written comprehension - The ability to read and understand 
information and ideas presented in writing. 

• CA2: Spatial orientation - The ability to know your location in relation to 
the environment or to know where other objects are in relation to you. 

• CA3: Selective attention - The ability to concentrate on a task over a 
period of time without being distracted. 

• CA4: Perceptual speed - The ability to quickly and accurately compare 
similarities and differences among sets of letters, numbers, objects, 
pictures, or patterns. The things to be compared may be presented at the 
same time or one after the other. This ability also includes comparing a 
presented object with a remembered object. 

• CA5: Memorization - The ability to remember information such as words, 
numbers, pictures, and procedures. 

• CA6: Mathematical reasoning - The ability to choose the right 
mathematical methods or formulas to solve a problem. 

• CA7: Deductive reasoning - The ability to apply general rules to specific 
problems to produce answers that make sense. 
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• CA8: Time sharing - The ability to shift back and forth between two or 
more activities or sources of information (such as speech, sounds, touch, 
or other sources). 

• CA9: Perceptual speed - The ability to quickly and accurately compare 
similarities and differences among sets of letters, numbers, objects, 
pictures, or patterns. The things to be compared may be presented at the 
same time or one after the other. 

• CA10: Oral expression - The ability to communicate information and 
ideas in speaking so others will understand. 

2. Physical Abilities 

Abilities that influence strength, endurance, flexibility, balance and coordination 

• PHA1: Extent flexibility - The ability to bend, stretch, twist, or reach with 
your body, arms, and/or legs. 

3. Psychomotor Abilities 

Abilities that influence the capacity to manipulate and control objects 

• PSA1: Finger dexterity - The ability to make precisely coordinated 
movements of the fingers of one or both hands to grasp, manipulate, or 
assemble very small objects. 

• PSA2: Manual dexterity - The ability to quickly move your hand, your 
hand together with your arm, or your two hands to grasp, manipulate, or 
assemble objects. 

• PS A3: Multi-limb coordination - the ability to coordinate two or more 
limbs (for example, two arms, two legs, or one leg and one arm) while 
sitting, standing, or lying down. It does not involve performing the 
activities while the whole body is in motion. 
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4. Sensory Abilities 

Abilities that influence visual, auditory and speech perception 

• SA1: Depth perception - The ability to judge which of several objects is 
closer or farther away from you, or to judge the distance between you and 
an object. 

• SA2: Near vision - The ability to see details at close range (within a few 
feet of the observer). 

B. SKILLS 

1. Basic Skills 

Developed capacities that facilitate learning or the more rapid acquisition of 
knowledge 

• BS1: Active learning - Understanding the implications of new information 
for both current and future problem-solving and decision-making. 

• BS2: Monitoring - Monitoring/Assessing performance of yourself, other 

individuals, or organizations to make improvements or take corrective 
action. 

2. Technical Skills 

Developed capacities used to design, set-up, operate, and correct malfunctions 
involving application of machines or technological systems 

• TS1: Quality control analysis - Conducting tests and inspections of 
products, services, or processes to evaluate quality or performance. 
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APPENDIX G. TASK EXPECTED FINAL STATES 



Figure 49. End State of Task 1. Source: (R. Lee, personal communication, March 

23,2018) 
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Figure 50. End State of Task 2. Source: (R. Lee, personal communication, March 

23,2018) 
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Figure 51. End State of Task 3. Source: (R. Lee, personal communication, March 

23,2018) 
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Figure 52. End State of Task 4. Source: (R. Lee, personal communication, March 

23,2018) 
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APPENDIX H. VIRTUAL OBJECT PLACEMENT 


The following figures contain white outline virtual objects presented to the subject during 
the experiment from an orthogonal view of top down. Printed sheets enabled virtual object 
locations measured in the real world, which aligned per design specifications. 



Figure 53. Virtual Object Guides for Task 1. Source: (R. Lee, personal 

communication, March 23, 2018) 
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Figure 54. Virtual Object Guides for Task 2. Source: (R. Lee, personal 

communication, March 23, 2018) 
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Figure 55. Virtual Object Guides for Task 3. Source: (R. Lee, personal 

communication, March 23, 2018) 
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Figure 56. Virtual Object Guides for Task 4. Source: (R. Lee, personal 

communication, March 23, 2018) 


116 















APPENDIX I. OUTLIER DATA 


Figures 57 through 63 depict raw data before removal of outliers. 


H0H 



-121674 


-S1116 -60837 


-20279 0 20279 


Figure 57. Efficiency: Absolute Error Original Histogram of Figure 6. 

3 *IQR = 31733.7 




Figure 58. Efficiency: Absolute Referential Error Original Histogram of Figure 10 

3 * IQR = 20254.59 
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Figure 59. Efficiency: Complexity Error Original Histogram of Figure 12 

3 * IQR = 80232.87 




Figure 60. Efficiency: Complexity Error (Placement Only) Original Histogram of 

Figure 14 3*IQR = 6301.35 
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Figure 61. Precision: Absolute Error Original Histogram of Figure 18 

3 */QK = 110.31 


• I—S3—I 



Figure 62. Time: Fong Part Original Histogram of Figure 28 3 * IQR = 12.06 
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Figure 63. Time: C Part Original Histogram of Figure 32 3 *IQR = 23.9 
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